Skip to content

Implementation evals: 69 eval-banana harness_judge design-conformance checks#42

Merged
fazpu merged 7 commits into
mainfrom
plans/implementation-evals
Jul 12, 2026
Merged

Implementation evals: 69 eval-banana harness_judge design-conformance checks#42
fazpu merged 7 commits into
mainfrom
plans/implementation-evals

Conversation

@fazpu

@fazpu fazpu commented Jul 9, 2026

Copy link
Copy Markdown
Member

What

plan/implementation_evals/ — the design-conformance acceptance suite for the final implementation: 69 eval-banana harness_judge checks, one YAML per check, each pointing an LLM judge at the binding sources (decision numbers + design docs) and at the code, with conjunctive conditions and a binary verdict. Discovered by eval-banana from eval_checks/; run with uvx eval-banana run --harness-agent ….

Coverage: D1–D63 + the requirements, across E0 documents (10), E1 blocks/chunks (2), E2 claims (9), E3 relations/observations (5), registries/ER/eval (5), K plane (4), projections (8), retrieval (10), ops/cross-cutting (8), code & boundary (8). Withdrawn decisions (D26–D30) and pure naming decisions excluded by design.

Conventions: absence = failure (acceptance for the complete system — filter by tags to match delivered phases); judge on a different model family than the producer (D53 applied to the eval itself); deliberate exceptions stated in-check so a correct implementation can't fail for obeying the design.

Codex review — applied in full

Per the process, Codex (gpt-5.5) independently reviewed the initial 53-check set (codex_review.md, included). It caught 4 factual errors — most notably my deletion check contradicted the binding schema (normal delete retains claims/evidence as audit history; only hard-forget scrubs) and my execution-classes check wrongly demanded ambiguity-scaled spend from E2 extraction (deliberately volume-proportional per D25/D31). All must-fixes applied; omnibus checks split for judge reliability (boundary→3, envelope→5, lineage→2); 9 checks added for its coverage gaps (D1/D46, D16, D22, D23, D36, code tooling, D62×2, D63); tags/cites fixed. All 69 validated against the eval-banana check schema.

Inconsistencies found while authoring (as requested)

Registered in questions.md §5:

(Also checked and found properly annotated: D37-vs-D51 raw-mount reversal, D12-vs-D45 hot-file removal, overall_design's K section.)

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown

Coverage report

This PR does not seem to contain any modification to coverable code.

fazpu and others added 3 commits July 10, 2026 00:30
… checks

plan/implementation_evals/ — acceptance checks that judge the final
implementation against the binding design (D1-D61 + requirements):
one YAML per check, self-contained instructions citing the binding
sources, conjunctive conditions, binary verdicts. Areas: E0 documents
(8), E1 blocks/chunks (2), E2 claims (9), E3 relations/observations
(5), entity resolution/review (3), K plane (4), projections (7),
retrieval (6), ops (6), code/boundary (3). All 53 validated against
the eval-banana check schema (unique ids, harness_judge, schema v1).

questions.md §5: register two inconsistencies found while authoring —
the requirements temporal-split paragraph predates D43/D49 (#30), and
the requirements E3 bullet omits observations (#31).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…16 checks (69 total)

Codex (gpt-5.5) reviewed the initial 53-check set (codex_review.md,
committed alongside). Applied in full:

- Factual must-fixes: deletion check now matches normal-delete
  retention semantics (claims/evidence kept as audit history; only
  hard-forget scrubs); execution-classes check names E2 extraction as
  the deliberate volume-proportional exception (D25/D31); Selection
  ledger check matches schema (plain keeps not ledgered); bitemporal
  check carries the D43 no-cap rule for measurements.
- Judgeability: split the omnibus checks (boundary D60 -> 3, retrieval
  envelope D49 -> 5, lineage/living D55 -> 2); reworded checks to
  demand concrete evidence (shared hydrator boundary, import-linter
  contracts, startup/CI family validator, targeted reuse tests).
- Coverage: +9 checks — D1/D46 source-of-truth split, D16 scope views,
  D22 golden sets, D23 registry scale, D36 sub-worker chain, code
  tooling/migrations, D62 delivery artifacts + enforced architecture,
  D63 embedding port config.
- Mechanics: missing decision tags added, cite paths fixed, README
  re-inventoried (69 checks, scope D1-D63).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…D64 (seed core) in the predicate check + README range

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@fazpu fazpu force-pushed the plans/implementation-evals branch from a51a98c to 878926d Compare July 9, 2026 22:32
fazpu and others added 4 commits July 11, 2026 16:31
…eview)

Every check now carries an explicit source-precedence rule in its
instructions (judges never see the README): the design docs are the
binding statement of the current system; decision entries are read
with their refinement/withdrawal annotations; on any apparent
disagreement the judge treats the designs (incl.
postgres_schema_design.md) as controlling and reports the discrepancy
rather than failing a design-conformant implementation. The 27 checks
that cited only decisions gain explicit design-doc citations. README
records the convention ("decisions for coverage, designs for truth").

Rationale: the decision log is append-only testimony refined by later
entries; authoring checks from raw decision prose risks failing a
correct implementation on superseded text — the exact hazard Codex's
review caught once (the deletion check, corrected from
postgres_schema_design §13.1).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…coverage, designs for truth)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Author review round 2: the precedence rule alone left the checks
decision-first in reading order (64/69 opened with "Read decisions.md
Dxx and <design>"). Flipped all of them: the Read instruction now
leads with the design doc / requirements section as "the binding
source", with decisions.md cited second for rationale and refinement
history. Also repaired ret_envelope_contradiction_completeness, whose
description had been corrupted by the earlier cite-insertion (its
description contained ", then", which the insertion script matched
instead of the Read sentence). Full sweep: 69 checks valid, none
decision-first, no polluted descriptions, every check cites a
design/requirements source and carries the precedence rule.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…hecks (76 total)

The round-2 review (codex_review_2.md, committed alongside) judged the
suite against the design docs as ground truth. All load-bearing quotes
were verified against the designs before applying. Applied in full:

- Surgery repairs: six mis-pathed requirements citations fixed
  (plan/requirements/ prefix) + binding-source phrasing added to the
  unflipped Read sentences; four non-design "binding source"
  designations rebound (CLAUDE.md, concepts.md, workers.md, the P3
  analysis are now context, not binding).
- Over-binding corrections (checks could fail conforming code):
  provider calls through D61 ports are conformant; direct-env access
  allowed via the requirement's reasoned per-line exception; ingest
  bumps legitimately cascade downstream; effective_period added to the
  claim_valid_kind enum; borderline per-claim grounding escalation is
  designed; observations use the shared adjudication_outcome enum;
  re-occurring relations in non-overlapping windows get new rows;
  reviewer may be a designated agent; writers run parallel across
  disjoint pages ("editing shared files" restored); model page binds
  writer invocations, not every page; version-vs-lineage tombstone
  branches spelled out; execution classes need an inspectable mapping
  (no invented enum); retry bound is a tunable per-stage default;
  producer/checker never-share per orchestration §8 (D53's
  recorded-exception prose loses under precedence); projection purity
  judged as outcome, not an invented import graph.
- D64 made scoreable (16-predicate seed incl. uses/reports_to
  signatures); D23 expanded to the full partitioned estate + the
  ungated load-test hook (a round-1 application miss).
- +7 coverage checks: K trigger surface/acyclicity, K eval canaries +
  citation faithfulness, K writer runtime two-band, K hard-forget git
  history, orchestration topology, S58 skill acceptance, continuous ER
  health.
- Tags reconciled with scored conditions; README recounted (76),
  column relabeled "primary binding sources", both reviews recorded.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@fazpu fazpu merged commit 19219d5 into main Jul 12, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant